<html>
<head>
    <title>Robots Exclusion Self Test</title>
</head>

<body>
<h1>Robots Exclusion Self Test</h1>
<p>This test needs to have the crawler read the robots.txt (TODO).</p>
<p>
    <ul>
        <li><b><a href="excluded.html">Excluded</a>: A page the robot is
            explicitly told not to visit.</li>
        <li><b><a href="excluded/level2/level3/excluded.html">Excluded
            Directory</a>: A page several levels down a directory structure
            the robot is told not to visit.</li>
        <li><b><a href="included.html">Included</a>: File to find.</li>
    </ul>
</p>
</body>
</html>
